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ABSTRACT 

California’s school-to-career (STC) efforts were examined in 
a statewide evaluation study that was initiated in December 2000 and 
scheduled for completion by June 2002. The study’s first phase was assessed 
in an interim report that focused on the following major activities: (1) 

development of a white paper describing the STC’s national and state status; 
(2) development of a statewide evaluation plan; (3) implementation of a 
statewide survey of employers and labor organizations involved in local STC 
activities; (4) establishment of working relationships with existing STC 
statewide committees and formation of a practitioners’ panel to advise the 
statewide evaluation effort; and (5) provision of assistance to the state in 
developing the local partnership (LP) case studies request for proposals and 
evaluation of the case studies proposals submitted by LPs . The highlights and 
lessons learned from each activity were summarized, and the insights gained 
during the interim evaluation were used to shape thinking about the final 
evaluation report’s structure and format and the following issues related to 
the methodology to be used in phase 2 of the study: (1) maximizing and 

documenting response rates; (2) applying common definitions to variables of 
interest; (3) implications of sampling for analysis across LPs, and (4) 
identifying the appropriate unit of analysis and reporting. (MN) 
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California School-to-Career Evaluation Study 
Interim Report 

I. Introduction 

School-to-Career (STC) represents a major education reform effort in California, 
designed to reinforce students’ academic learning through real-world and work-related 
contexts and to help students draw the connections between what they currently do in 
school with their future education and career aspirations. In order to gauge the progress 
and impact of STC as a statewide education reform strategy, California’s governor, in 
collaboration with the STC Interagency Partnership (lAP) consisting of the 
Superintendent of Public Instruction, the Chancellor of the California Community 
Colleges, and EDD, contracted with WestEd and MPR Associates to conduct a statewide 
evaluation of STC in California. This statewide evaluation study is required by the 
School-to-Work Opportunities Act of 1994 (STWOA), under which federal funds are 
provided to California to support development of its STC system. 

Underway since December 2000, the statewide evaluation study is scheduled for 
completion by June 2002. Four major research questions, listed in Table 1 below, guide, 
the evaluation. 



Table 1. Research Questions to Guide the Evaluation 

(1) What is. the status of STC implementation in California? 

c: How many schools, teachers, and students are participating in STC activities? 

c Are there regional differences or other discernible patterns in the type or degree of 
STC participation? 

(2) How has STC affected student preparation for post-secondary education and 
career entry? 

What types of students are being influenced by STC for post-secondary and 
employment opportunities? What types of students are not being influenced? 

o What is the evidence that students who participate in STC are better prepared for 
post-secondary education and/or career entry? 
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(3) To what degree and in what ways has STC contributed to systemic change? 

What changes in school programs or infrastructure (e.g., new models of career 
development, new career-focused curricula, new partnerships with employers to 
provide work-based experiences for students) are associated with or can be 
reasonably attributed to STC efforts? 

o How are local and state STC initiatives coordinating with other education reform 
and assessment initiatives? 

(4) Have STC principles penetrated the community deeply enough to be 

sustainable? 

o How will services and activities supported by STC funds be supported by other 
sources after STWOA funding ends? 

o Do stakeholders see value in sustaining a STC system? 

o How will existing STC systems function after the end of state and federal 
support? 



Phase 1 evaluation activities have been completed, and Phase 2 evaluation activities are 
just now underway. At this mid-point juncture in the study, it is valuable to review the 
work completed to date, and its implications for the final evaluation report. 



Purpose of Report 

This Interim Evaluation Report summarizes preliminary evaluation findings and 
discusses ongoing methodological and data collection strategies, including any 
adjustments to the original evaluation study design. The primary focus is on progress to 
date, rather than on summative outcomes of the evaluation. It is premature to focus 
heavily on outcomes of the evaluation, given that key aspects of the study are still 
underway. Nevertheless, by documenting the work in progress, the Interim Evaluation 
Report serves to foreshadow the final evaluation report that will be submitted in Summer 
2002. 



Structure of the Report 

The Interim Evaluation Report is organized as follows. Section II describes the 
highlights of Phase 1 activities. Section III provides an overview of Phase 2 activities. 
This is followed, in Section IV, by a discussion of methodological issues and challenges 
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for Phase 2. Finally, Section V describes how the Final Evaluation Report will be 
organized. 

II. Review of Phase 1 Evaluation Activities and Lessons Learned 
Overview of Phase 1 Evaluation Activities 

Phase 1 consisted of a series of activities and accomplishments that set the course for this 
important statewide evaluation study. Specifically, Phase 1 activities served three key 
functions. First, they established a solid anchor for Phase 2 evaluation activities by 
defining California’s baseline STC status and creating the working relationships and 
networks necessary to conduct a comprehensive statewide evaluation study. At the same 
time they provided important stand-alone products, specifically the White Paper and an 
Employer/Labor Organization Survey, which were of immediate use to the state and 
Local Partnerships (LPs). Finally, Phase 1 activities piqued interest and began the 
momentum in the field for the statewide evaluation and its potential benefits. 

The accomplishments for Phase 1 can be summarized under five major categories of 
activities: 

t: develop a White Paper to describe the national and state status of STC; 

c: develop a statewide evaluation plan; 

c: conduct a statewide survey of employers and labor organizations who are 

participating in local STC activities; 

c: establish working relationships with existing STC statewide committees and 

convene a Practitioners Panel to advise the statewide evaluation effort; and 

t: assist the state in developing the LP Case Studies Request for Proposals and 

evaluate Case Studies proposals submitted by LPs. 

The highlights and implications associated with each of these activities are described 
below. 
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Highlights and Lessons Learned from the White Paper 

The first major deliverable in Phase 1 was the White Paper. Authored by WestEd and 
MPR project staff, the paper examined and summarized the current status of STC in 
California. In preparing this comprehensive paper, project staff researched and 
s)mthesized information from a variety of sources, including: LP applications for 
STWOA funds; LPs’ Quarterly Reports; data from two federally-sponsored longitudinal 
evaluations of STWOA implementation (the National Evaluation of STW 
Implementation and the STW Progress Measures); and existing evaluations conducted for 
LPs. The draft White Paper was submitted to the lAP on April 17, 2000. After receipt of 
comments, the final White Paper was formally submitted on July 7, 2000 and then 
formatted for Web site posting on August 21, 2000. 

Major Findings from the White Paper 

In conducting the research on the White Paper, project staff discovered substantial 
amounts of existing data about STC in California. However, these data are based on few 
common measures or reporting formats, are primarily qualitative, and focus on 
processes/plans implemented by LPs rather than on student outcomes. Given that LPs 
are, for the most part, still engaged in the process of building systems and programs, the 
limitations of existing data are understandable. However, current data about STC in 
California do not provide information that is of the highest interest to funders and the 
Legislature (e.g., student performance data), nor do they allow us to directly answer key 
research questions (e.g.. What impact does STC have on students ’ transition to post- 
secondary education?). 

Despite these limitations, existing data on California’s STC efforts to date, as outlined in 
the White Paper, show the following: 

The diverse goals of STC efforts in California suggest that both attitudinal and 
performance measures are needed to assess the full influence of STC on students, 
teachers, employers, and systems. 

t: On a statewide basis, California maintains school-level, not student-level data 

about student academic performance; without student-level data it is impossible to 
evaluate the influence of STC on student performance. 
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c As evidenced in individual LP STC implementation plans, there is a lack of 
common concepts and vocabulary, exacerbating evaluation difficulties associated 
with a lack of common measures across LPs. 

c: Historically, California's response rate to STW-related surveys, and therefore the 

availability of key comparable quantitative, data, has been notably low. 
Specifically, California’s response rate on the 1997-98 Progress Measures Survey 
was 57%. Similarly, the response rate for 1998-99 was 56%.’ 

Available data on STC implementation in California lack specificity with respect 
to either quality or intensity of STC activities. 



In summary, the White Paper concluded that currently available data do not allow for 
strong inferences about the status of STC in California relative to sustainability and 
student preparation, or how well LPs are meeting their goals. However, while existing 
data have important limitations, particularly with respect to STC impact and student 
outcomes, meaningful information on STC process and implementation were gleaned 
from them. These findings informed the evaluation design for Phase 2 activities and are 
described below. 

Implications of White Paper Analyses for Phase 2 Activities 

Overall, White Paper analyses of STC implementation, data quality, and data availability 
indicated that Phase 2 evaluation activities must strike a balance between (1) canvassing 
LP activities across the state to achieve a broad understanding of California’s STC 
progress and impact, and (2) delving deeper into a more limited number of LPs to better 
understand systemic change and sustainability of STC. Additional implications included 
the following: 



1. Diverse STC implementation strategies suggest the need for some flexibility in 
data collection as well as common protocols/measures for Phase 2 evaluation 
activities. 

2. Diverse implementation strategies aiming for impact at multiple levels (student, 
teacher, employer, local system, state system) necessitate evaluation at a variety 
of levels. 



’ The most recent Progress Measures Survey (1999-2000) had a higher response rate (86%) than in previous years; 
however, this is most likely attributable to a much shorter survey than in previous years as well as extensive telephone 
follow-up which was not conducted in other years. 
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3. Because many STC activities aim to change students’ attitudes as. well as skills 
and knowledge, attitudinal measures are appropriate in conjunction with 
performance and other outcome measures. 

4. Seeking correlations between STC activities and changes in student performance 
or attitudes in places where STC activities have been minimal (e g., one Career 
Day or one job shadow over the course of a year) will yield minimally useful 
information. Therefore, evaluation activities need to focus on LPs where STC 
activities have been substantial. 

5. While there has been limited implementation of complex school-based and work- 
based activities, we must focus on these activities, as they have the greatest 
chance to influence student outcomes. 

6. Given that existing data offer little to no comparability, standardized, well-defined 
indicators must be developed. Similarly, standardized, well-defined data 
collection instruments and methods are necessary. 

7. Given that many LPs, particularly new ones, are attempting to build systems 
rather than isolated programs, evaluation of system change is critical. 



Highlights and Lessons Learned from the Employer/Labor Organization Survey 1 

The Employer/Labor Organization Survey was a statewide survey concerning the roles 
employers and labor organizations are currently playing in California’s STC effort, 
factors affecting participation, and recommendations to increase involvement. It was 
conducted between March 12 and June 15, 2000 by WestEd/MPR project staff. 

LPs provided contact information for employer and labor organizations that were 
participating in local STC activities and those who were invited to participate, but did 
not. They submitted contact information for approximately 12,500 employers and 150 
labor organizations. From this population, a stratified sample was created, and the survey 
was mailed to more than 2,300 employers and 102 labor organizations statewide. Project 
staff conducted telephone follow-ups to increase the response rate. Approximately 19% 
of the employers returned useable survey forms, as did almost 24% of the labor 
organizations. At the request of LPs, information identifying the employers and labor 
organizations that participated in the survey is being kept confidential. (See Appendix A: 
Employer/Labor Organization Survey 1.) 
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Recommendations Based on Survey Results 

WestEd/MPR staff made the following recommendations based on the findings of the 

statewide STC evaluation survey.^ 

t: Employers overwhelmingly identified two factors that contributed to their company’s 

decision to participate in STC: (1) promoting student awareness of specific 
careers/industries and (2) providing realistic expectations of work. Therefore, LP 
marketing efforts to increase employer participation should consider emphasizing 
how employer involvement in STC can promote student awareness of specific careers 
and provide students with realistic expectations of work. 

r: Responding employers believe that internships, job shadowing, speaking in classes, 

and mentorships are the best ways to influence students. Therefore, emphasizing 
these activities should be considered in employer engagement efforts. 

^ Non-profit organizational employers are the least involved in STC activities. Over 
25% of non-profit respondents stated that they were not familiar with the activities of 
their LPs. The most common barriers to participation by non-profits were: 
unfamiliarity with STC activities; lack of available resources; time and energy 
required for participation; and possible liabilities. More than one-third of the non- 
profits stated that wage subsidies and financial incentives would significantly increase 
their participation, and almost 40% found the time and energy required to 
communicate with schools and STC partnerships a barrier to participation. To 
increase participation from the non-profit sector, LPs might target engagement efforts 
specifically at non-profits and address the barriers of communication. However, at 
this time, it may not be possible to overcome the financial barriers inhibiting non- 
profit participation. 

^ Survey data suggest that larger employers participate in larger numbers of STC 
activities. In addition, public sector employers were somewhat more likely than 
private, for-profit or non-profit organizations to participate in larger numbers of 
activities. Given limited program resources, the most efficient strategy to increasing 
employer involvement may be to pursue larger employers and public sector 
employers, both of which have a relatively high likelihood of STC participation. 

c. Smaller companies appear to be relatively under-represented in STC activities. Thus, 
our recommendation was that outreach activities include assessing how to reach 
smaller employers. 



^ Note that because the LPs provided lists of employers and labor organizations for the survey, the survey sample 
should not be constmed as representative of all of California. Nevertheless, the survey’s results allow for identification 
of some general trends and recommendations about increasing STC participation in the future. 
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c: Employers responded that they most frequently had STC relationships with their local 

high schools versus middle and elementary schools (66% versus 37% and 38%, 
respectively).^ However, high schools comprise only 12% of the public schools in the 
state while more than 73% are elementary schools, and 15% middle schools.'* If the 
STC initiative is to be a comprehensive effort, the relationship between employers 
and middle and elementary schools must be strengthened. 



Labor Participation in STC 

Labor organizations are not yet major players in local STC efforts. Relatively few LPs 
have deeply engaged partners from labor organizations and when involved, labor 
organizations appear concentrated in few industries. Based on these findings, major 
efforts are needed to engage them in STC activities. 

Working with STC-related Groups 

A major assumption underlying the statewide evaluation design is that critical . 
stakeholders need multiple opportunities and avenues (e.g., face-to-face meetings, email, 
and conference calls) to provide input and feedback on evaluation activities and 
deliverables. This is especially true since stakeholders have responsibilities for providing 
the actual data for the statewide STC evaluation. During Phase 1, project staff used 
several avenues for communication and information with STC-related groups regarding 
the statewide evaluation. 

Informing the lAP and California STC Committees of the Evaluation Work 
As might be expected at the initial stage of any large, statewide effort, the evaluation 
project director, Daniel McLaughlin, maintained daily contact with the lAP contact, Chris 
Rury, during the first 13 months of the project. In this way, the LAP was kept informed 
about project developments as they occurred, including any needs for adjustments in the 
evaluation workplan or schedule. At a more formal level, project staff provided quarterly 



^ Sums to more than 100 percent due to multiple responses. 

^ Education Data Partnership (EdData) (On-line). Available http://www.ed-data.kl2.ca.us/dev/State.asp 
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progress reports to the lAP. Moreover, project staff submitted outlines, drafts,, and final 
products for lAP review. 

WestEd/MPR staff also made project presentations at the request of the lAP; Several of 
these presentations took place at project onset to inform key groups about the design of 
the study and to elicit their feedback. Specifically, project staff made a presentation to 
the Evaluation and Accountability Committee on January 1 1, 2000 in Sacramento. In 
February 2000, WestEd/MPR senior staff attended a California STC Advisory Council 
meeting in San Diego to provide an overview of the statewide study. In March 2000, 
WestEd/MPR made a presentation to EDD senior staff in Sacramento to introduce the 
evaluation study. In April 2000, Daniel McLaughlin participated in the California 
delegation to the Marketing/Evaluation conference in San Antonio. The purpose of this 
event was to coordinate and share effective strategies for marketing and evaluation 
efforts. 

Other presentations were connected with specific project deliverables. For example, 
WestEd/MPR staff members made a presentation on the findings of the Employer/Labor 
Organization Survey 1 to the Employer/Labor Involvement Committee on July 12, 2000. 
WestEd/MPR also provided an update about the evaluation to the System Accountability 
Committee on July 19, 2000 and delivered a presentation about the evaluation at the 2000 
STC Summer Academy. 

Establishing and Convening the Practitioners Panel 

Besides the LAP, the LPs are another constituency from which project staff sought regular 
input. Our first step was to invite all LP directors to participate in a March 14, 2000 
telephone conference. During this telephone conference, project staff “walked” 
participants through a Web-based slide show that introduced the design and components 
of the statewide STC evaluation plan. We also answered questions raised by LP directors 
and representatives about the statewide STC evaluation study. 
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The next step was to establish a Practitioners Panel in Spring 2000 to provide a forum for 
ongoing LP feedback on evaluation design, data gathering, and instruments. While the 
majority of panel members are LP directors and evaluators, other members include an 
employer representative, an lAP staff member, and a member of the Evaluation and 
Accountability Committee. 

The Practitioners Panel consists of two groups: (1) a Steering Committee of five 
members who have committed to work closely with LAP and WestEd/MPR staff, and (2) 
a Technical Review Committee of approximately ten members who have committed to 
review evaluation work. The first meeting of the Practitioners Panel was held on April 
14, 2000 at WestEd’s San Francisco offices. Prior to this meeting. Practitioners Panel 
members were provided with several documents to review, including: draft criteria for 
selection of LPs for the Case Studies; a draft outline for the STC Evaluation White Paper; 
and a draft of the Employer/Labor Organization Survey 1 . The actual meeting focused 
on the overall statewide evaluation design and elicited participants’ feedback on the draft 
documents. 

Subsequent meetings of the Practitioners Panel were held by telephone conference. In 
November 2000 and January 2001, telephone conferences were held in order to obtain 
Panel member feedback on draft instruments (e.g., surveys and interview protocols) that 
project staff were developing for the Case Studies. Most recently, the outline for this 
Interim Evaluation Report was sent to the Practitioners Panel for review and comment on 
March 30, 2001. 

Supporting the LP Case Studies Request for Proposals (RFP) and Review Process 
As part of the statewide evaluation of STC, California set aside substantial funding (up to 
$2.5 million) for case study evaluation efforts to be conducted by selected LPs. Designed 
by WestEd/MPR project staff, the LP Case Studies are intended as in-depth 
investigations of selected local STC efforts to evaluate: local implementation progress to 
date; the influence of STC involvement on student outcomes; the contribution of STC to 
systemic change; and the potential for sustaining local STC systems. Case Study efforts 
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conducted by LPs will also be highly useful to those LPs for their own evaluation and 
sustainability efforts.^ The 61 current and former State/Federal STC implementation and 
Urban Rural Opportunity (UROG) grantees were invited to submit a proposal, with the 
anticipation that 1 5 to 18 would be selected through the competitive bidding process. 

Developing the LP Case Studies RFP 

WestEd/MPR staff worked closely with lAP staff to develop the LP Case Studies RFP, 
whereby interested LPs were invited to submit a competitive bid for Case Study funding. 
Specifically, project staff drafted substantial parts of the RFP, with the first draft 
submitted to the lAP for review in August 2000. Between August and October, 
WestEd/MPR worked closely with lAP staff to revise and refine the RFP, and maintained 
regular communication with LPs to apprise them of progress on the RFP and the schedule 
for RFP release. The RFP was posted on the state Web site on October 26, 2000. 

Following the release of the RFP, WestEd/MPR and LAP staff conducted a technical 
assistance workshop on November 9, 2000 in Sacramento to present the RFP and answer 
questions from LPs interested in developing a proposal. Subsequent to this meeting, 
WestEd/MPR and lAP staff twice posted written questions and answers about the RFP on 
the state Web site. (See Appendix B: Questions and Answers from Technical Assistance 
Workshop.) 

Selecting Case Study School Samples 

One of the primary goals of the California STC evaluation is to identify the influence on 
students' academic achievement of participating in STC-type school reforms. To 
accomplish this, LPs conducting in-depth Case Studies were required to identify schools 
that have had substantial STC implementation and those that have not. Defining these 
"high STC implementation" schools is necessarily a somewhat subjective process, 
because LPs, school districts, and individual schools have been encouraged to implement 
STC systems and activities in ways that meet their own local needs. However, LPs were 



5 While evaluators were hired by LPs to conduct evaluation activities at respective sites, for the purpose of this report, 
LP refers to the party conducting the evaluation. 
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asked to use existing data and knowledge to make judgements about high schools 
meeting or not meeting the criteria detailed in the RFP. (See Appendix C: Appendix H of 
Case Study RFP.) 

On November 30, 2000, all LPs interested in submitting a Case Study proposal were 
required to provide a comprehensive list of high schools in their LP to WestEd/MPR, 
identifying each school as either “high STC implementation” or “other.” A total of 20 
LPs submitted lists, ranging in size from 1 to 58 high schools. In December 2000, 
WestEd/MPR staff worked intensely to process the lists submitted by LPs and to generate 
stratified random samples of schools for each LP. This labor-intensive process entailed 
drawing multiple samples for several LPs, as replacements were sought for selected 
schools that elected not to participate in the study. Each sample resulted in 
overrepresentation of high schools with high STC implementation. Such oversampling 
ensures sufficient representation of high schools with robust STC activities, and at the 
same time, allows for generalizing of findings across a given LP. (See Appendix D: List 
of Selected Sample High Schools.) 

Designing and Participating in the Proposal Evaluation Process 
As LPs worked on Case Study proposals, project staff assisted lAP staff in designing the 
process for training of proposal reviewers and in the actual scoring of proposals. To best 
address the range of STC models of governance and implementation strategies in 
California, WestEd/MPR proposed key sample characteristics and selection criteria for 
the Case Study sample. WestEd/MPR proposed that the overall sample be diverse, with 
representation from different geographic regions, types of locales (urban, suburban, 
rural), sizes of LPs, and populations targeted for participation. (See Appendix E: Details 
of Key Sample Characteristics.) Given the key role of the Case Studies to the overall 
statewide evaluation plan, project staff worked closely with the LAP and obtained input 
from the Practitioners Panel to establish the criteria for selecting LPs for the Case 
Studies. 
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Proposals were due on January 3, 2001. Soon after, project staff examined all 
submissions in order to select examples for the purpose of training proposal reviewers. 
Three WestEd/MPR staff attended and assisted in conducting the proposal review session 
that took place on January 9 through 1 1, 2001 in Sacramento. Fifteen LPs submitted 
completed proposals and the competitive process resulted in 1 3 LPs selected to conduct 
Case Studies as part of the overall statewide STC evaluation effort. (See Appendix F: List 
of LPs Applying for Case Study Funding, List of LPs Awarded Case Study Funding, and 
Description of Populations Served by Awarded LPs.) 

Summary of Phase 1 Findings 

In summary, there are a number of salient findings or outcomes from Phase 1 that have 
clear implications for next steps in the study. First, the inconsistent quality of existing 
data on STC was underscored during our research on the White Paper. Clearly, there is a 
need for uniform quality indicators in order to properly ascertain the status and progi'ess 
of STC in California. Second is the apparent interest in the field for reliable STC data to 
help inform state and local planning and implementation efforts. Many LPs expressed 
this interest to us as we posted the White Paper and designed the Case Studies. Third, 
WestEd/MPR staff were able to apply our state and national experience to play an 
advisory role to LAP staff in STC matters that have implications beyond specific 
evaluation activities. The level of our involvement with LAP staff during Phase 1 was 
both professionally satisfying and necessary to the progress of the study. Finally, the 
information gleaned from Phase 1 activities clearly position California to maximize the 
potential of and better understand the possible obstacles to Phase 2 evaluation 
implementation activities. 
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III. Overview of Progress on Phase 2 Evaluation Activities 



How Phase 2 Activities Build from Phase 1 Activities 

Evaluation activities planned for Phase 2 build directly on what was undertaken during 
Phase 1. First, using the findings of the White Paper to define the baseline status of STC 
activities highlighted the complexity of the California STC landscape, and the 
unevenness in the availability, format, and quality of existing data about STC in 
California. Therefore, the evaluation design in Phase 2 takes into account the need to 
purposefully triangulate different evaluation design and data sources. This will help to 
mitigate shortcomings in the data from any one source through the collection of similar 
evidence by multiple methods and from other data sources. As Table 2 shows, the 
evaluation research questions of interest are addressed through a variety of data collection 
strategies and instruments. 

Table 2: Primary Data Collection Strategies/Instruments 
to Address STC Evaluation Research Questions 



Research Question 


Primary Data Collection 
Strategy/Instrument* 


1 . What is the status of STC 
implementation in California? 


c CORE Case Studies Interview 
Protocols (administrators, teachers, 
counselors, employers, labor 
representatives) 
c Employer/Labor Survey 
c LP Director Telephone Interviews 
c STC Evaluation White Paper 


2. How has STC affected student 
preparation for post-secondary 
education and career entry? 


^ CORE Senior Survey 
^ CORE Senior Follow-up Survey 
c PLUS Case Studies 


3. To what degree and in what ways has 
STC contributed to systemic change? 


^ CORE Case Studies Interview 
Protocols 

c LP Director Telephone Interview 
Protocol 


4. Have STC principles penetrated the 
community deeply enough to be 
sustainable? 


c Employer/Labor Survey 
c LP Director Telephone Interview 
Protocol 

^ CORE Interview Protocols 



* The selected strategies and instruments identified in Table 2 for each research question 
were designed to be the primary means for answering a particular question. 
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Second, findings from the first administration of the Employer/Labor Organization 
Survey in Phase 1 will inform the design and administration of a second Employer/Labor 
Organization Survey to be undertaken during Phase 2. 

Third, in Phase 1, WestEd/MPR established working relationships with existing STC 
statewide committees, and organized a Steering Committee and Practitioners Panel to 
serve in an advisory capacity. During Phase 2, our work with these key stakeholders is 
continuing as they offer feedback about Case Study instruments, drafts of reports, etc. 
Finally, in Phase 1, WestEd/MPR assisted in the development of an RFP for Local 
Partnership Case Studies. In Phase 2, project staff are designing surveys, interview 
protocols, and reporting formats for these Case Studies, as well as monitoring the 
evaluation activities undertaken by LPs and offering technical assistance. 

Components of the Phase 2 Evaluation 
Telephone Interviews of all LP Directors 

In the original evaluation design, a plan to develop and administer a survey to all LP 
directors was proposed. The survey would address key aspects of each of the four major 
research questions, as well as those not adequately addressed by existing surveys, such as 
the state rollout strategy and activities, coordination of state and local resources, and the 
incorporation of state and national skill standards into local STC. 

While surveys are typically a cost-effective and efficient method for collecting data, the 
usefulness of the resulting data depends on achieving an adequate response rate. One of 
the notable findings reported in the White Paper is California’s low response rate on 
STW-related surveys. This finding, coupled with our own experience with similar 
surveys, led us to reconsider administering the LP director survey and to consider 
alternative means of data collection from LPs throughout the state. 

Instead of mail surveys, our revised plan is to conduct telephone interviews of all LP 
directors who are in non-fimded LPs. The LP Director Phone Interview will be brief, and 
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will consist of a strategic mix of closed and open-ended questions that will provide a 
combination of qualitative and quantitative data about potential changes in LP funding, 
structure, and implementation after STWOA sunsets. These interviews will be conducted 
in the fall of 2002. 

Although more time- and labor-intensive, telephone interviews typically result in better 
response rates. WestEd/MPR’s goal is to focus efforts on achieving a 90% response rate 
for the phone interview by implementing several follow-up strategies, including 
additional phone calls and letters. Since project staff will concentrate resources on 
realizing this high response rate, only one telephone interview of LP directors will be 
conducted rather than two mail surveys, as proposed in the White Paper. Through this 
change in data collection strategy, we hope to obtain a more complete picture from LP 
directors of STC sustainability. 

Synthesis of Data from Extant Sources 

During Phase 2 of the evaluation, project staff will continue to examine existing sources 
of data for evidence related to this evaluation’s questions of interest. Three sources of 
data for this aspect of the study include the National School-to-Work Progress Measures 
Survey, narrative quarterly reports completed by LPs for the state, and evaluation reports 
conducted by LPs not participating in Case Study research.® 

Initially, project staff hoped to analyze the longitudinal data provided by these sources. 
However, it is important to note that the 1999-2000 Progress Measures Survey is 
significantly different than Progress Measures Surveys administered in previous years. 
Specifically, the 1999-2000 survey has fewer items and also asks questions that had 
previously been asked through Mathematica’s Local Partnership Survey. These changes 
were made to reduce the amount of time needed by respondents to answer the survey, 
with the hope that, as a consequence, response rates would increase. The change in 



® An anticipated fourth data source was the Local Partnership Survey, conducted by Mathematica Policy Research. 
However, contrary to expectations, that survey was not administered in fall 2000. Instead, the National School-to-Work 
Office requested that more extensive data be collected from state-level School-to-Work directors and a small sample of 
Local Partnerships. 
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survey format will have implications for longitudinal data analysis during Phase 2, since 
many of the key questions asked in previous years are no longer on the 1999-2000 
version of the survey. (See Appendix G: 1999-2000 Progress Measures Survey.) 

Also, as was noted in the White Paper, LPs’ quarterly reports to the state generally are 
not consistent with each other in style or format, and they do not include quantitative or 
comparable measures. Consequently, from these reports we cannot draw inferences about 
activity levels within or across LPs, nor can we draw conclusions about progress toward 
achieving goals. However, we will use extant data to supplement the findings from our 
ongoing data collection efforts (e.g., the Employer/Labor Organization Survey, the LP 
Director Interviews, and the Case Studies). Analyses of extant data will help us make 
meaningful comparisons and examine growth or other trends of STC activity within and 
across LPs. 

Employer/Labor Organization Survey 2: Approach and Link to Previous Survey 
As per the original REP for the California STC Evaluation Study, the evaluation plan 
calls for administering two Employer/Labor Organization Surveys: the first in May 2000 
in Phase 1, and the second in Spring 2001 during Phase 2. As was described earlier, the 
first survey was mailed as planned in May 2000, and results were presented in a report 
entitled “Results and Recommendations based on the Employer/Labor Organization 
Survey” in September 2000. 

Upon completion of the September 2000 report describing results of the first survey, the 
STC Employer and Labor Involvement Committee expressed interest in revising plans 
for the survey’s second administration. Specifically, concerns were raised that 
administering a second svuvey less than a year after the first survey administration would 
provide few new insights. Moreover, Committee members were concerned that the 
sample of employers responding to the first survey was limited and therefore might not 
be representative of the employers and labor organizations throughout the state. 
WestEd/MPR staff asked for feedback regarding the second administration of the survey 
at a meeting on July 12, 2000. However, the Employer and Labor Involvement 
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Committee has not yet offered recommendations to WestEd/MPR about how to proceed, 
and survey administration has been delayed. In order to help the Committee in the 
decision making process, WestEd/MPR staff described possible approaches for the 
Committee’s consideration, and detailed the benefits and drawbacks of each approach. 
The desirability and feasibility of each option varies. These options are presented below. 

1. Random Statewide Survey of All Employers and Labor Organizations 

On the face of it, a logical approach would involve a random sample of all employers and 
labor organizations throughout the state. This would yield a comprehensive picture about 
how employers and labor organizations across the state view STC. Hovvever, this 
approach depends on the use of an existing and readily accessible data file for employers 
and labor organizations, respectively. As described in the White Paper, available project 
resources will not allow for the creation of these databases specifically for this project, 
nor are project resources sufficient to conduct substantial reformatting, manipulation, and 
error checking and resolution on an existing statewide database system. Moreover, 
surveying a sample of all employers in California may not yield as useful information as 
targeting those employers that have familiarity and can provide infonnation about what 
works and does not work from an informed perspective. Therefore, WestEd/MPR feel 
this option is neither feasible nor advisable. 

2. Follow-up to First Survey 

One approach proposed in the White Paper is to use the second survey administration as a 
follow-up to the first survey administration. In this way, the focus would be on the sub- 
population of employers and labor organizations that have some exposure and familiarity 
with STC. Specifically, a two-pronged survey approach for employers was proposed: 

( 1 ) re-survey the actual respondents to the first survey and (2) survey another random 
sample of employers from the existing project database of employers. For labor 
organizations, the following was proposed: (1) re-surVey respondents to the first survey 
and (2) survey additional labor organizations identified by the LPs involved in the Case 
Studies. 
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Although this option is preferable to the first in terms of efficiency and targeting an 
informed sub-population, it also has its limitations. Specifically, as noted by the STC 
Employer and Labor Involvement Committee, the focus on examining progress over less 
than one year’s time by surveying the same respondents twice may not yield the most 
useful results. In addition, identifying the very same individual respondents from the 
initial survey would be problematic because in many instances, LPs supplied names of 
businesses and organizations rather than the names of contact people. Thus, there is no 
guarantee that the same individuals who completed the first survey could be reached for 
the follow-up survey. 

3. Sustainability Survey to Employer/Labor Organizations as Identified by LP Case 
Study Sites 

This option calls for a new, succinct survey that focuses primarily on sustainability— a 
major research issue for Phase 2. As such, the survey would: (1) clarify or delve more 
deeply into issues uncovered from the first survey, arid (2) prompt employers and labor 
organizations to provide their perspective on whether they believe STC is sustainable and 
what it would take to sustain it. This approach specifically targets the 13 LPs funded to 
conduct Case Studies and asks them to identify employers and labor organizations with 
which they have been in contact concerning STC. There are advantages associated with 
targeting the 13 Case Study LPs. First, these 13 LPs are already conducting evaluation 
research with WestEd/MPR, so that providing information on employers and labor 
organizations should pose little additional burden on them. In contrast, we found it 
difficult during the first Employer/Labor Organization Survey to obtain usable 
employer/labor contact information from some LPs. Specifically, many LPs that were 
not engaged or interested in evaluation research chose not to provide lists, or provided 
incomplete or informally written lists. As it stands, project staff already have usable lists 
of employers and labor organizations from 1 1 Case Study LPs that participated in the first 
survey. Thus, we would only need to ask these 1 1 LPs to update their contact lists and 
then work with the other two to supply initial lists. In summary, the advantages of this 
option are that it is focused on a specific key issue (sustainability) and targets a fairly 
accessible sub-population that is geographically spread across 13 different LPs. 
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4. Employer/Labor Organization Focus Groups on Sustainability 

This option would forego a survey altogether and instead rely on focus groups to provide 
information from select employers and labor organizations. WestEd/MPR staff would 
conduct regional focus groups in Northern California, Southern California, and the 
Central Valley. The advantage of this approach is that it allows for dynamic group 
discussion and potentially richer information. The disadvantage is that we would reach 
fewer individuals and organizations than through a survey. Moreover, another STC- 
related project may be planning to conduct employer and labor organization focus groups 
in the near future, which may result in some overlap of effort. 

5. Redirect Survey Funds to Provide More Technical Assistance to Case Study LPs 
Although much work has been accomplished during Phase 1 , the bulk of our statewide 
evaluation work lies ahead. The longitudinal Case Studies, which will be conducted 
during Phase 2, are a key component of the overall evaluation. As will be discussed in 
detail below, during the early stages of Phase 2, WestEd/MPR recognized that a number 
of LPs will require a much higher level of technical assistance than was anticipated in the 
original workplan. Given the importance of the Case Studies to the overall evaluation 
design it is imperative that LPs receive the additional support they need from 
WestEd/MPR. We strongly recommend that the LAP consider redirecting funds 
originally earmarked for the Employer/Labor Organization Survey 2 to providing 
additional technical assistance to Case Study LPs. 

LP Case Studies 

The unique contribution of the Case Studies to the overall evaluation design is that they 
provide an opportunity to collect complex, in-depth information. It is primarily through 
concerted study of selected LPs that the evaluation can glean detailed information and 
insight about the inner workings of STC and the nuances of systemic change. Most 
importantly, the Case Studies will allow the state to closely examine the ways in which 
STC sustainability is being achieved. 
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Description of Case Studies Approach 

The Case Study design was significantly influenced by the knowledge gained during 
Phase 1 evaluation activities. The following considerations assisted in shaping the 
longitudinal study: 

1. To best understand STC implementation across the state, the evaluation needs to 
focus on LPs where STC activities have been substantial. Little can be gained by 
focusing on LPs that have been unable to create or sustain a viable STC program. 

2. Given the diversity of STC implementation strategies used by the LPs, the Case 
Study design would have to provide flexibility in data collection. Moreover, since 
LP strategies aim for impact at multiple levels (student, teacher, employer, local 
system, and state system), evaluations would have to be conducted at various 
levels. 

3. Attitudinal measures must be examined in conjunction with student performance 
indicators since many STC activities aim to change students’ attitudes as well as 
skills and knowledge. 

4. Existing data on STC offer little to no comparability. Therefore, standardized, 
well-defined indicators, data-collection instruments, and methods linked to these 
indicators must be developed. 

Based on these factors and WestEd/MPR’s goal of acquiring detailed information on LP 
strategies for implementing and sustaining STC activities across the state, the following 
Case Studies approach was developed. 



CORE and PLUS Evaluation Activities 

Case Study activities are divided into two sets: CORE and PLUS. Applicant LPs had the 
option of applying for CORE funds only or for both CORE and PLUS funding. CORE 
activities are central to the Case Studies. They build on existing STC evaluation 
resources (e.g., the Progress Measures and Local Partnership Surveys) as a basis for 
describing STC implementation and assessing the potential for sustaining these education 
reforms. 

CORE evaluation efforts focus primarily on activities and people associated with high 
schools (students, teachers, employers, labor organizations, etc.), but also address K-8 
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schools and community colleges to a lesser extent. This high school focus is consistent 
with the evaluation’s desire to identify arenas where STC activities are delivered to large 
numbers of students with relatively high implementation. High schools are generally the 
most likely place to find large numbers of students receiving multiple STC experiences of 
longer duration. 

PLUS evaluation activities focus specifically on the question: How has STC participation 
affected student preparation for postsecondary education and career entry? Interested 
LPs could design their own specific PLUS research questions and evaluation activities as 
long as their PLUS activities (1) focus exclusively on the relationship between STC 
implementation and student performance measures, and (2) use quantitative analyses to 
answer the targeted research question. 

To conduct CORE and PLUS evaluation activities, LPs were required to secure the 
services of a qualified evaluator or to demonstrate that existing internal evaluation 
resources would meet the demand of completing the workplan (see Appendix H: CORE 
Case Study Workplan). Evaluators play a crucial role in Case Study data collection and 
analysis. Therefore, LPs were expected to sufficiently demonstrate that evaluators’ 
experience and expertise would enable them to meet the challenges of the rigorous 
workplan. The qualifications of evaluators were heavily weighted during the proposal 
scoring process. 

Training the LP Directors and Evaluators 

Because CORE Case Studies aim to gather and analyze comparable data from a diverse 
set of LPs, CORE activities are highly prescribed. All LPs awarded CORE funding must 
use the data-gathering instrumentation provided by WestEd/MPR over specified periods 
of time and present analyses in formats prescribed by WestEd/MPR. The use of common 
instrumentation and formats for analyses is designed to ensure standardization across 
cases and make the evaluation process easier and more efficient for LPs. 
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Given the importance of standardization across all CORE Case Studies, WestEd/MPR 
staff held a two-day training for LP Directors and Evaluators on February 13-14, 2001 to 
discuss required protocols, reporting formats, and other issues associated with the Case 
Studies. This training was designed to be the priihary means of providing technical 
assistance to the LPs. 

A total of 45 individuals attended the training, including representatives from each of the 
Case Study LPs, state-level employees, and project staff. The training agenda included 
an overview of the key elements of the Case Study design, a discussion of contractual 
issues and requirements, sampling issues, and quarterly and final reports. The project 
statistician provided instruction on calculating analysis weights, selecting samples, and 
determining and maximizing response rates for the Senior Survey. Other topics of 
discussion included interview protocols, school site visits, the Local Partnership 
Administrator Survey, and the timeline for completion of Case Study data collection and 
analysis. (See Appendix I: Training Day Agenda and Materials.) 

Although the primary focus of the two-day training was on CORE evaluation activities, 
LP Directors and evaluators had opportunities throughout the training to ask questions 
related to PLUS evaluations. Moreover, an optional questiomand-answer session was 
held for PLUS evaluators at the end of Day 2 of the training for those needing additional 
technical assistance. 

Developing LP Data Collection Instruments and Analysis Protocols 

All CORE data collection instruments for Case Studies were developed by WestEd/MPR, 

with input from the Practitioners Panel, and were designed to provide LP evaluators with 

both quantitative and qualitative data that address the evaluation’s four main research 

questions. 

CORE Case Study research entails administering the following instruments (See 
Appendix J; Examples of all Available Instruments): 



ERIC 
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c: Administrator Surveys (Fall 2001') . These surveys will be given to principals of all 

public K-12 schools. Since elementary, middle, and high schools typically have 
different approaches to STC implementation, slightly different versions of the survey 
have been developed for each of these levels. Topics include: partnership 
composition and governance, career awareness and development activities offered by 
partnership schools, career-related programs of study, work-based learning strategies, 
secondary-postsecondary linkages, standards and certification, professional 
development, and strategies for the future. CORE evaluators are expected to achieve a 
70% response rate on these surveys. Evaluators will report data to WestEd/MPR that 
are aggregated across all schools completing the survey. 

L' Senior Survey . In April 2001, CORE evaluators from each funded LP will administer 
a two-page Scantron survey, developed by WestEd/MPR, to a stratified, random 
sample of 150 seniors from the class of 2001 from each high school in the CORE 
intensive sample. Topics on this survey include students’ participation in various STC 
activities, their attitudes about STC, and their career and educational plans after 
graduation. In addition, the survey includes demographic items such as students’ 
ethnicity and their parents’ educational levels. Evaluators are expected to achieve an 
80% response rate at each school. Data from each of the school sites will be 
aggregated across each LP. To ensure that each LP reports comparable data to 
WestEd/MPR from this survey, project staff provided LPs with “data shells” that 
clearly specify how data must be analyzed. 

L' Senior Follow-Up Survey . In Fall 2001, respondents who completed the first senior 
survey will be asked to complete a second survey. Evaluators will be expected to 
achieve a 75% response rate on this instrument. Because WestEd/MPR wants the 
design of the survey to be informed by the results of the first survey, this instrument 
has not yet been developed. Overall, the purpose of this instrument is to determine if 
the initial plans seniors described in the first survey were realized. This follow-up 
component is generally not conducted because of the logistical difficulties of locating 
students after graduation. However, data from this survey will allow WestEd/MPR to 
ascertain if graduates are taking the first steps in meeting their educational and career 
goals. As was the case with the first senior survey, LPs will be provided with detailed 
data shells to ensure comparable data analysis across LPs. 

o School Site Interviews (Spring-Fall 2001) . At each high school identified as a CORE 
intensive site, evaluators will conduct a series of interviews and/or focus groups of 
counselors, principals, other administrators (e.g., STC Coordinator), teachers, and key 
school site staff. The purpose of these interviews is to get a comprehensive picture of 
the implementation of STC from data aggregated across each of the CORE intensive 
schools. Although LPs are not required to submit school-level data to WestEd/MPR, 
project staff created a school-level reporting format that will enable evaluators to 
provide each high school with data about the status of STC at their site, using 
information from the school site interviews, both senior surveys, and the 
administrator survey. 

l: LP Director Interview ('Summer 20011 . Each evaluator will conduct an extensive 

interview with the LP director, covering topics such as LP composition and 
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governance; LP relationships with employers, community-based organizations, labor 
organizations, and other stakeholders; the influence of STC on academic standards, 
career awareness and development, curricula, and assessment; and efforts to sustain 
and expand STC efforts. 

c Emplover/Labor Organization Interviews (Suniuner 200 U . From each high school in 
the LP’s CORE intensive sample, evaluators will be asked to identify and interview at 
least five employers of students who participated in career majors or academies, as 
well as all labor organizations working with students who participated in pre- 
apprenticeships or apprenticeships. Topics covered on the interview protocol include: 
roles of the employer/labor organization in STC efforts, sustainability of STC, and the 
influence of STC on students’ preparation for emplo 3 mient and transition to 
postsecondary education. 



Strategies for Supporting and Monitoring the LP Case Studies 
Prior to begirming the evaluation work, several LPs expressed questions and concerns 
regarding various aspects of the Case Study evaluation process. At the initial trainitig, 
and in subsequent phone calls and email messages, LPs articulated a need for additional 
assistance in implementing various aspects of the evaluation, including administering the 
Senior Survey — a critical data gathering component. Initially, it was intended that 
WestEd/MPR would develop the evaluation instruments and guidelines for the Case 
Studies, provide training, monitor LP implementation progress, and synthesize 
information across the Case Studies! Moreover, WestEd/MPR agreed to be available for 
technical assistance when necessary. However, based on the questions expressed by the 
LPs, it became clear that ongoing guidance, interventions, and in-depth technical 
assistance would be required during the entire evaluation process. It also became 
apparent that, if LPs were to provide consistent and accurate data, WestEd/MPR’s role 
would need to intensify considerably from primarily a monitoring entity to include that of 
a technical assistance provider. In response to LP requests, WestEd/MPR have created 
various materials and resources to guide LPs through the evaluation process. Materials 
include a vignette detailing school-level data analysis for a fictitious high school and 
prescriptive reporting formats. (See Appendix K: Vignette and Reporting Format.) 



While LPs expressed the need for these frequent and informative communications, 
limited evaluation resources made it impossible to sustain such a level of intervention. 
Therefore, WestEd/MPR developed the STC Case Studies Forum, a Web-based means of 
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communicating with LPs, sharing resources, and providing technical assistance. 

Designed in collaboration with WestEd’s Director of the Application Development 
Group, the Forum is email and Web-based, allowing project staff to make general 
announcements and providing LPs the opportunity to ask targeted questions regarding 
their evaluations. Unlike a Listserv, which is generally facilitated, the Forum is 
moderated by project staff, ensuring that LP questions are answered in an accurate and 
timely manner.^ The Web-based feature allows participants to view questions and 
responses in their entirety. In addition, questions posed by LPs will provide 
WestEd/MPR staff insight on the progress of the evaluation to inform decisions and 
strategies for interventions as necessary. 

The Forum also provides a means for disseminating instruments and materials during the 
various stages of the evaluation. All materials are posted in various formats which can be 
downloaded across platforms. The Forum is the primary means of providing LPs with 
technical assistance and will be adapted in response to the changing needs of the LPs. 

Review of LP Quarterly Report and Draft Final Reports 

To ensure that the various stages of the evaluation are progressing accordingly, LPs are 
required to submit detailed quarterly reports throughout the evaluation. This is a 
purposeful strategy for monitoring the progress of each LP and to ensure that project staff 
can provide additional guidance or interventions if and when necessary. Sequential 
quarterly report requirements include: 

c: Quantitative analyses of data gathered from the Senior Survey in April 2001. 

(Reporting formats are provided.) 

c: A final narrative description and analyses of data gathered from the Survey 

administered to high school seniors in April 2001, as well as analyses and 
narrative description of data gathered from interviews with employers, labor 
organizations and LP directors. (Reporting formats are provided.) 

Quantitative analyses and narrative of data gathered from Senior Follow-up 
Survey; data from all interviews; and quantitative analysis and narrative of data 



^ In this instance, “facilitated” refers to a passive, “hands-off’ approach to technically managing an on-line discussion. 
“Moderated” refers to a more active role where the moderator has a much greater level of responsibility for maintaining 
the integrity and content of the on-line discussion. 
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for Administrator Surveys of all K-12 public schools conducted in September- 
October 200 1 . (Reporting format will be provided.) 

^ Final synthesized analyses and narrative of all required data. (Report outline will 
be provided.) 

Timeline for Phase 2 Evaluation Activities 

As demonstrated by the information required in these periodic and prescribed reports, 
data requirements are distributed to allow for a logical sequence in data collection and 
build on prior work conducted. In addition, when developing the CORE timeline, project 
staff took into consideration schools’ calendars in order to minimize the burden on 
schools and LPs. 
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IV. Phase 2 Methodology Issues Related to the Final Report 

WestEd/MPR have brought together a diverse set of quantitative and qualitative 
approaches to answer the primary research questions in Phase 2. The specific approaches 
for Phase 2 activities were designed within the context of the state’s interest in a 
comprehensive statewide evaluation of STC activities and the constraints of conducting 
analyses in a large state within a fairly short time period. The research design also takes 
into account the additional requirements imposed on the use of federal STW funds. 
Finally, the design is intended to be sufficiently flexible to accommodate any 
unanticipated challenges or changes associated with the complex STC landscape in 
California. 

While the Phase 2 approaches and activities have been described at length in both the 
White Paper and in Section III of this document, we would like to emphasize key 
methodological issues that are likely to affect the final evaluation results. This section 
focuses on four methodological issues that are emerging as key to analysis and reporting 
of results for the final evaluation report. They include: maximizing and documenting 
response rates, applying common definitions to variables of interest, implications of 
sampling for aggregating results, and identifying the appropriate unit of analysis. 



Maximizing and Documenting Response Rates 

Surveys represent a major strategy for data collection across different components of this 
evaluation. WestEd/MPR project staff rely on surveys to collect information from 
employers and labor organizations. The Case Study LPs are using surveys to collect 
information from high school students (Senior Survey, Senior Follow-up Survey) and 
school administrators (Elementary, Middle, and High School Surveys). Although 
surveys constitute an efficient way to collect information on a large number of possible 
respondents, the usefulness of the resulting data depends on achieving an adequate 
response rate. Therefore, maximizing response rates is a critical concern of this study. 
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As previously described in Section II, the potential for an inadequate response rate led us 
to reconsider one of our Phase 2 activities: a mail survey to the directors of all LPs across 
the state. The original intent of the survey was to determine the status of STC penetration 
in California LPs, with particular emphasis on the question of STC sustainability. Instead 
of mail surveys, our revised plan is to conduct telephone interviews of LP directors. 
Although more time intensive, telephone interviews typically result in better response 
rates. Through this change in data collection strategy, we hope to achieve a higher 
response rate and, hence, a more complete picture from LP directors of STC 
sustainability. 

Despite the proposed change in strategy for obtaining data from all LP directors, 
switching from surveys to interviews is not appropriate or feasible when hundreds or 
thousands of potential respondents are targeted. For example, such switches are not 
appropriate for the CORE LP Case Studies, as they were specifically designed to include 
a strategic balance of surveys and interviews. Even so, selected LPs have expressed 
concerns about achieving the high response rates for the student and administrator 
surveys called for in the REP. In response to these concerns, a substantial amount of time 
was spent on possible strategies for increasing survey response rates during the initial 
February 2001 training of Case Study LPs. Since LPs continue to ask for technical 
assistance in this regard, LPs that are successful in obtaining a high response rate are 
encouraged to share their effective strategies on the emailAVeb-based Case Study 
Discussion Fomm. 

In summary, the issue of maximizing response rates is likely to be a concern throughout 
the study. However, both WestEd/MPR project staff and individual LPs are monitoring 
and documenting the response rates achieved for purposes of the final report. 



Applying Common Definitions to Variables of Interest 

Another notable White Paper finding is the lack of common STC definitions and 
concepts across the state, as evidenced by the different terminology and meanings 
described in individual LP STC implementation plans. Given this finding, project staff 
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investigated and provided common definitions of variables of interest for different Phase 
2 data collection activities. For example, definitions for “high intensity” STC schools 
and students were provided to LPs conducting Case Studies. LPs were instructed to use 
the definition of “high intensity” (or “high STC implementation”) schools to categorize 
all the schools in their LP (as “high intensity or “other”). MPR staff then used the 
categorized lists provided by LPs to draw stratified sarhples. 

Our experience to date is that merely providing common definitions is not enough to 
ensure uniform application of these definitions to variables of interest. Specifically, 
many LPs that are conducting Case Studies continue to raise questions on a daily basis 
about the definitions we have provided for high intensity schools and students, and some 
want to tailor the common definitions or use their own. With the recent “roll out” of the 
Case Studies Senior Survey, we are also finding differences across schools in the use of 
STC-related terms such as “career academy” and “career pathway.” As such, we have 
advised all LP evaluators to be cognizant of how the schools in their sample are using 
these terms so that the survey administrators can clarify any potential confusion about 
terminology for students before they complete the surveys. Local evaluators will be 
advised to make similar clarifications before administration of the administrator surveys. 

In summary, the lack of consistency in how STC-related terms have been used across the 
state is a challenge that we are currently facing in our evaluation implementation. We 
have handled this issue proactively in the Case Studies by advising all LPs to use the 
common definitions of “high intensity” students and schools to re-examine their 
classifications. If LPs found some schools to be inadvertently misclassified, we 
instructed LPs to reclassify them for purposes of analysis. During our synthesis of 
findings across LPs, WestEd/MPR will also take into account any reclassification or 
differences in defining of variables of interest in reporting on STC trends and patterns. 

Implications of Sampling for Analysis Across LPs 

Well-conducted sample surveys allow us to estimate the incidence of STC participation 
and attitudes about further schooling and career opportunities in the larger population 
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from the sample of students, employers, and labor organizations contacted in Case Study 
research. Surveys provide uniform question wording, known probabilities of selection, 
and the ability to generalize within well-defined limits.* Probability sampling allows us 
to aggregate the information across selected schools to provide partnership-wide 
estimates for all high school seniors in an LP. It also allows us to estimate how much 
variability there is in our LP-wide estimates due to sampling, and thus the size of 
“confidence” intervals around estimated means and proportions. 

Although the sampling methods used to conduct the Senior Survey are based on 
probability sampling, LP results cannot be combined together to provide statewide 
estimates for all high school seniors. Although we can aggregate across high schools 
within an LP to provide an LP-wide estimate, we cannot do the same across LPs.^ 

All the reasons cited in the footnote below for not aggregating data across LPs apply not 
only to the Senior Survey, but to all other data collection methods used in the Case 
Studies. Therefore, the final report will provide major findings broken out by each LP, 
will describe trends and patterns across LPs, but will not combine data to provide 
“statewide” estimates. Since part of the LP selection included a desire to reflect factors 
such as geographic (North and South) diversity, urbanicity, and student diversity, we 
believe the results will provide a comprehensive picture of STC activity as it occurs 
across the state. 



* In the CORE Case Studies, for example, the Senior Survey is based on a stratified random sample of all high schools 
served by a LP, Schools were selected according to strict sampling procedures, with each school having a known 
probability of selection. Students within schools were also selected using a stratified random sample based on the 
intensity of their exposure to STC activities, 

^ The reasons for this are reasonably straightforward. First, not all STC LPs participated in the case studies. Of the 61 
LPs invited to submit a proposal, only 15 submitted responses to the Case Study RFP. This introduces a “selection” 
effect, meaning that there is likely to be something different about those LPs that wrote proposals compared with those 
that did not, and this difference is likely associated with important aspects of the study. Second, not all the LPs that 
submitted proposals were funded, thus introducing another selection element. Third, although all the CORE studies 
follow a uniform approach to sampling and student selection, the actual implementation varies from school to school 
within LPs because the schools themselves vary in idiosyncratic ways. Within LPs, we expect local evaluators will 
design a consistent approach to handling the differing situations they find. Their decisions about how to deal with local 
conditions are not necessarily the same as the decisions that another local evaluator in a different LP would make. 

Thus, variations among evaluators introduce another uncontrolled element that argues against combining results. 
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Identifying the Appropriate Unit of Analysis and Reporting 

Because this statewide evaluation encompasses disparate LP Case Study evaluation 
efforts that must be compiled and synthesized, it is important to make explicit for each 
component of the study the appropriate unit of analysis'® and reporting. For example, the 
analysis protocols for the CORE studies have been designed to identify the appropriate 
unit of analysis, thereby helping to ensure comparability of analyses and reporting across 
LPs. Furthermore, the protocols are intended to encourage local evaluators to synthesize 
the different views of STC that the quantitative studies (e.g., the Senior Survey and the 
Senior Follow-up Survey) and the qualitative studies (such as interviews with teachers, 
school administrators, local employers, and LP directors) provide. (As previously noted, 
the specific partnership-level reporting requirements are attached as Appendix K.) 

In order to provide a comprehensive and meaningful picture of STC, the appropriate unit 
of analysis for each Case Study is at the partnership level. Indeed, the LP is the 
appropriate focus for this study because this is the level at which funding and major 
decision-making regarding STC occurs. Moreover, with over 120 schools across the Case 
Studies, it is beyond the scope and resources of this statewide evaluation to analyze and 
report results for individual schools. 

Although the individual school is not the targeted unit of analysis for the Case Studies, 
participating schools are asked to provide a wealth of data. These schools are motivated 
to participate in the statewide evaluation, in large part, by interest in their own school- 
level data that is generated from this study. Indeed, many local LP evaluators have 
agreed to provide schools with this data. Therefore, project staff have provided a school- 
level reporting format to local evaluators as part of overall technical assistance. Local 
evaluators can use this format to focus their school-level analyses and reports in a form 
that is both useful for their own analysis peeds as well as for providing data to schools. 



'® A unit of analysis is the persons or things being studied, and can be individuals, groups, organizations, etc. Choosing 
the appropriate unit of analysis is important since this decision has implications for how data can be generalized. 
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V. 



Structure of The Final Evaluation Report 



Our analysis of the progress to date on the statewide evaluation study has helped shape 
our thinking about how to organize the final evaluation report. Moreover, the proposed 
organization for the final evaluation report is based on key assumptions about how to 
maximize the usefulness of the report. Specifically, we assumed that the final evaluation 
report must: 

1 . provide clear and direct answers to the research questions of interest about the 
status of STC implementation in California, how STC affects student preparation 
for the future, how STC contributes to systemic change, and the sustainability of 
STC; 

2. fill the existing data gaps in our knowledge about STC by including 
comprehensive and comparable data from sites across California; 

3. provide meaningful data syntheses, interpretation, and recommendations to help 
inform next steps and policies concerning STC in California; and 

4. be designed to accommodate widespread dissemination and use by different 
constituents. 



With these criteria in mind, we propose to organize the final evaluation report into six 
main sections, followed by appendices. The report will also include a non-technical 
executive summary that will be suitable for wide dissemination. Each main section of the 
report is described below. 



Section I: Introduction 

This section will introduce the study, its purpose, and the historical and socio-political 
context in which it was designed and implemented. The introduction will also present the 
research questions of interest and lay the groundwork for the rest of the report. 

Section II: Methodology 

In this section, the overall evaluation design and methodologies will be described. It will 
also present the rationale for the various methodologies, how they supplement and 
complement each other, as well as how they address the research questions of interest. 
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Section III: Results of Statewide Evaluation Components 

This will be the first of three sections on evaluation results. This section will present the 
findings for the statewide evaluation components (i.e., non-Case Study components), 
including: the telephone interviews of the LP directors, the Employer/Labor Organization 
Survey 2, and the syntheses of data from extant sources (e.g.. National School-to-Work 
Surveys, LP quarterly reports, other evaluation reports). 

Section IV: Cross-LP Analyses of CORE Case Studies 

This section will present the trends and patterns that are discerned from the cross-LP 
analyses and syntheses of CORE Case Studies. To facilitate greater understanding and 
usability of these results by LPs and other interested audiences, this section will present 
findings in a similar format to the one we designed for individual LPs to use in reporting 
Case Study partnership level data. Specifically, we will present findings in the following 
major categories: 

Partnership composition, funding, and roles and responsibilities 
o Career awareness, exploration, and development activities 
Career academies and career majors/pathways 
o Curriculum integration strategies 
Work-based learning 

o Principals, teachers, and professional development 
c New standards and certification 

o Connections with business, labor organizations, and postsecondary institutions 

o Connections between STC participation and outcomes for students involved in 
STC 

o Sustainability of STC efforts 
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Section V: Cross-LP Analyses of PLUS Case Studies 

Whereas the previous section presents cross-LP findings of CORE Case Studies, this 
section will focus on the findings of PLUS Case Studies. As such, it will highlight trends 
and patterns across LPs concerning the relationship between student 
performance/achievement and STC participation (as measured by test scores, grade point 
averages, etc.). 



Section VI: Conclusions and Recommendations Regarding STC in California 
Based on results across all components of the statewide evaluation, this section will draw 
conclusions regarding STC in California and make specific policy recommendations. We 
will also distill the important “lessons learned” about STC. The conclusions and 
recommendations will be organized around the four research questions of interest: status 
of STC implementation; STC effects on student preparation for the future; STC 
contribution to systemic change; and sustainability of STC. 
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